Implement lua-vec - vectors as native Lua datatype #1728

Pirulax · 2020-10-17T15:07:01Z

This PR aims to fix the issue we have with userdata vectors. (#321)
As discussed on discord (and maybe even in the issue itself), using Lua tables would be only marginally better, and the performance benefit would probably be lost when doing table <=> conversion.
Note: I've tested this with bytecode as well (Compiled all default resources, and they all worked perfectly). So no backwards compatibility issues here.

This code(Lua-vec) was "stolen" from here.

I need opinions: Although I've already discussed this with @Woovie, I'd be great if others could give their feedback.
I think naming the datatype as vector would be totally okay (the function table will be named vector in lower case, just so we keep Lua's naming convention. We can just alias it to Vector in a form of a global variable)

Features in lua-vec:

Nice indexing of vectors, either with indecies, .x, or even .xxyz, .xx (like in hlsl)
Really fast (even compared to Lua table implementation). from 4x up to 10x faster than Vector4.
Very memory friendly. Our current Vector implementation is a memory hog([1]). Lua-vec uses no additional memory at all! Its free! [2]

[1]: A Vector4 needs 4 bytes for lua_newuserdata, another sizeof(unsigned long) + sizeof(void*) + sizeof(CLuaVector4D (which is 16 + 4 bytes) in CIdArray
[2]: Its value is stored in a GcObject union. This is where the userdata is stored as well, so, compared to the userdata implementation, we use no additional memory

Todo:

Make vec callable (add __call metamethod), right now vectors can only be created with vec.new
Make it possible for ArgumentParser and CScriptArgReader to read the new type as a Vector2/3/4D
Add support to construct Vector2/3/4 from native vector
Add support to construct native vector from existing Vector2/3/4 (only way to implement this cleanly is to use .x .y .z)
@qaisjp suggested that it would be nice to have a separate branch to keep track of our modifications to Lua.
Implement LUA_TVEC in CLuaArgument (WriteToBitStream and ReadFromBitStream)

Tests:
Resource used: vectest.zip
Scene rendering with smallpt.lua (from Lua-vec):

Command: runsmallpttests 250 250 4 false

Settings: w: 250, h: 250, samples: 4, do_force_gc_collect: false
============[vec]============
Writing to file... took 176 ticks
Tracing took 115081 ticks
Memory used: 3.3 gb

============[stdlua]============
Writing to file... took 291 ticks
Tracing took 371085 ticks
Memory used: <= 3.3 gb

============[Vector4]============
Ran out of memory. Needs around 16 gigs.

Command: `runsmallpttests 250 250 4 true`

Settings: w: 250, h: 250, samples: 4, do_force_gc_collect: true
============[vec]============
Collect garbage total: 19830 ticks
Writing to file... took 165 ticks
Tracing took 103906 ticks

============[Smallpt - Vector4]============
Collect garbage: 25859 ticks
Writing to file: 338 ticks
Tracing: 475007 ticks

============[Smallpt - stdlua]============
Collect garbage: 15044 ticks
Writing to file: 344 ticks
Tracing: 360707 ticks

Syntethic add/create/mul/sub benchmark:

Iterations: 10000000
============[native-vector - GC: OFF]=======
=> create: 2694 ms
=> sub: 1692 ms
=> mul: 1681 ms
=> eg: 535 ms
=> add: 1716 ms
============[native-vector - GC: ON]========
=> create: 2228 ms
=> sub: 1338 ms
=> mul: 1313 ms
=> eg: 507 ms
=> add: 1327 ms
===========================================

============[Vector2 - GC: OFF]============
=> create: 9966 ms
=> sub: 10291 ms
=> mul: 10085 ms
=> eg: 2138 ms
=> add: 9660 ms
============[Vector2 - GC: ON]============
=> create: 30965 ms
=> sub: 21562 ms
=> mul: 21275 ms
=> eg: 5635 ms
=> add: 17737 ms
===========================================


============[Vector3 - GC: OFF]============
=> create: 22923 ms
=> sub: 11093 ms
=> mul: 12490 ms
=> eg: 2196 ms
=> add: 13755 ms
============[Vector3 - GC: ON]============
=> create: 47758 ms
=> sub: 15084 ms
=> mul: 15346 ms
=> eg: 4736 ms
=> add: 13413 ms
===========================================

============[Vector4 - GC: OFF]============
=> create: 14349 ms
=> sub: 31588 ms
=> mul: 24745 ms
=> eg: 2147 ms
=> add: 12531 ms
============[Vector4 - GC: ON]============
=> create: 50227 ms
=> sub: 16102 ms
=> mul: 15704 ms
=> eg: 4504 ms
=> add: 13207 ms
===========================================

This reverts commit a1801c6.

Pirulax · 2020-11-13T05:05:14Z

So, whats people's opinion on this?
I'd be really happy to see this in 1.6.
Also it seems like we could without problems add support for matrices as well. A GCObject is 180 bytes, a regular 3x3 matrix is.. 48, which means that the unions size would stay the same.

ghost · 2020-12-09T10:57:24Z

I like the changes.

@qaisjp suggested that it would be nice to have a separate branch to keep track of our modifications to Lua.

We can have a submodule for Lua. In this way, it will be easy to track the changes.

Pirulax · 2020-12-09T13:02:51Z

So, we should create a separate repo?

ghost · 2020-12-09T15:06:37Z

So, we should create a separate repo?

Hmm, that doesn't seem like a good idea. Let's just create a new branch for Lua. I'd do it like this:

Download Lua 5.1 original source and commit the code to the branch.
Copy the Lua source from MTA vendor to the new branch.
And then create a PR for lua-vec changes.

qaisjp · 2020-12-09T17:39:50Z

My approach would be:

Fork Lua repo to multitheftauto org. This keeps original history instead of destroying the Lua 5.1 history, and makes it easier to cherry-pick improvements later.
Cherry-pick our custom 5.1 mods from mtasa-blue to a branch on the Lua repo
Replace vendor/lua in mtasa-blue with a submodule that targets that Lua repo
Create a PR for lua-vec changes in the Lua repo.

This approach is compatible with the submodule system proposed in #319.

Pirulax · 2021-03-08T14:17:19Z

AFAIK There is no official Lua repo on GitHub, if that's what you meant by Fork Lua repo.
I dont think the Lua history is necessary. I'll just create a new repo with the 5.1 source code from Lua website

Pirulax · 2021-03-08T15:51:45Z

Okay I think I did it.
Is this what you meant?

AlexTMjugador · 2021-03-13T11:09:17Z

AFAIK There is no official Lua repo on GitHub, if that's what you meant by Fork Lua repo.
I dont think the Lua history is necessary. I'll just create a new repo with the 5.1 source code from Lua website

There's an official mirror on GitHub of the Lua repo, here. It has a tag for every released Lua version, all the way back to 5.1. I agree that keeping the history may come in handy in the future for some cherry-picks, for updating Lua more easily, or whatever, but I would also understand not keeping the original commit history if it's deemed too verbose.

Pirulax · 2021-03-14T20:51:42Z

Please look at MTA org's /lua repo.
I have filtered our commits to vendor/lua + added base Lua at the top (source code for 5.1.5 downloaded from Lua website).
Would that fit?

LosFaul · 2021-03-16T20:30:38Z

i noticed you call the vector type in Lua only "vec" which is inconsistent to the other types: "number", "table", ...
the type should be "vector"

LosFaul · 2021-03-16T20:36:55Z

since Lua uses doubles, shouldn't the vectors not use doubles too?

Pirulax · 2021-03-16T20:41:45Z

There would be no benefit of the additional memory required. MTA uses float vectors everywhere, the additiona precision would be lost.
Yeah, the type name will be changed, once this PR gets reviewed or something.. Tbh, we didnt really agree on a name yet. I think vector (perhaps with a capital V) is acceptable. Since it's not vanilla Lua, we might as well stick to our naming convention.

LosFaul · 2021-03-17T11:13:45Z

maybe cVector
where c stands for custom
or if someone has better idea he should write it

LosFaul · 2021-03-17T11:15:58Z

btw. since my knowledge is not too big about instruction sets
so, only asking if it's possible:
could it be possible that the vectors could somehow utilize something like sse2 for even higher performance?

AlexTMjugador · 2021-03-17T12:51:14Z

Please look at MTA org's /lua repo.
I have filtered our commits to vendor/lua + added base Lua at the top (source code for 5.1.5 downloaded from Lua website).
Would that fit?

I'm not sure if you were talking to me, but yeah, it looks good to me. I was just pointing out that the original approach that @qaisjp suggested was possible due to the existence of that mirror.

Pirulax · 2021-03-17T13:19:24Z

Anyways, the submodule stuff #2112

Pirulax · 2021-03-17T13:23:33Z

The main bottleneck is probably still the Lua side, rather than the actual maths. Not worth the headache. But where I think we might benefit from SSE is our vector stuff.. although I wonder why MSVC doesn't use SSE by default there.. Should be pretty trivial I belive.

…into feature/native-vectors

Pirulax · 2021-03-23T21:31:17Z

Edit: Turns out there's 2 lua-vec branches: One which is a first-class value like a number, bool, etc.. and the second one (this) treats it as GCObject's. Obviously the first one is faster, but comes at the expense of making TValue be 16 bytes instead of 8 (Which translates to basically 2x memory usage for Lua values) and probably performance loss as it would take 4 memory fetches to move the value into the register. The GC'd version is probably already fast enough IMHO. Although it uses some hackery (some kind of free list?).

botder · 2023-07-25T14:36:11Z

@Pirulax A great amount of time has passed, but do you still want to get this pull request merged?

Nico8340 · 2024-04-27T04:27:31Z

This is very useful and would be a good update.
You don't plan to continue dealing with this PR in the future?

YSAFE · 2025-01-20T16:06:43Z

Bump?

Rilot06 · 2025-04-25T12:02:13Z

Any news on this? Would make my current project way easier and faster instead of countless float math operations each time.

Pirulax · 2025-10-13T21:04:26Z

I think it can be merged, no?

Edit: Actually nvm, this is just the veclib, i'd still have to change all the parser stuff and whatnot to return this type instead of Vector3...
I don't really have the time to deal with this PR to be honest.
So, if anyone has the time to work on it, then great, we can get this merged... Otherwise it is what it is.

Pirulax added 5 commits October 17, 2020 06:17

Copy-paste merge Lua-Vec into vendor/lua

005dfdc

Add lua_openlib to CLuaMain

bc90295

Temporary hacked code for smallpt.lua test

a1801c6

addendum to 005dfdc

3d97042

Revert "Temporary hacked code for smallpt.lua test"

3fb0552

This reverts commit a1801c6.

qaisjp self-requested a review October 17, 2020 15:40

StrixG added the enhancement New feature or request label Oct 17, 2020

StrixG added this to the Backlog milestone Oct 17, 2020

StrixG added the hacktoberfest-accepted label Oct 17, 2020

Pirulax changed the title ~~Implement lua-vec - vectors are lua datatype~~ Implement lua-vec - vectors as native Lua datatype Oct 18, 2020

qaisjp removed this from the Confirmed Issues milestone Dec 7, 2020

StrixG removed the hacktoberfest-accepted label Dec 30, 2020

Merge branch 'master' of https://github.com/multitheftauto/mtasa-blue …

03f39a2

…into feature/native-vectors

Merge branch 'master' into feature/native-vectors

9e2bb99

TheNormalnij mentioned this pull request Jan 2, 2024

Luau #2815

Closed

Uh oh!

Implement lua-vec - vectors as native Lua datatype #1728

Are you sure you want to change the base?

Implement lua-vec - vectors as native Lua datatype #1728

Uh oh!

Conversation

Pirulax commented Oct 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pirulax commented Nov 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Dec 9, 2020

Uh oh!

Pirulax commented Dec 9, 2020

Uh oh!

ghost commented Dec 9, 2020

Uh oh!

qaisjp commented Dec 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pirulax commented Mar 8, 2021

Uh oh!

Pirulax commented Mar 8, 2021

Uh oh!

AlexTMjugador commented Mar 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pirulax commented Mar 14, 2021

Uh oh!

LosFaul commented Mar 16, 2021

Uh oh!

LosFaul commented Mar 16, 2021

Uh oh!

Pirulax commented Mar 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LosFaul commented Mar 17, 2021

Uh oh!

LosFaul commented Mar 17, 2021

Uh oh!

AlexTMjugador commented Mar 17, 2021

Uh oh!

Pirulax commented Mar 17, 2021

Uh oh!

Pirulax commented Mar 17, 2021

Uh oh!

Pirulax commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

botder commented Jul 25, 2023

Uh oh!

Nico8340 commented Apr 27, 2024

Uh oh!

YSAFE commented Jan 20, 2025

Uh oh!

Rilot06 commented Apr 25, 2025

Uh oh!

Pirulax commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Pirulax commented Oct 17, 2020 •

edited

Loading

Pirulax commented Nov 13, 2020 •

edited

Loading

qaisjp commented Dec 9, 2020 •

edited

Loading

AlexTMjugador commented Mar 13, 2021 •

edited

Loading

Pirulax commented Mar 16, 2021 •

edited

Loading

Pirulax commented Mar 23, 2021 •

edited

Loading

Pirulax commented Oct 13, 2025 •

edited

Loading